Skip to content

Updating EDOT collector configuration samples for mOTLP input #10569

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 14 commits into
base: main
Choose a base branch
from
Open

Updating EDOT collector configuration samples for mOTLP input #10569

wants to merge 14 commits into from

Conversation

gizas
Copy link
Contributor

@gizas gizas commented Oct 15, 2025

  • Docs

What does this PR do?

Updates the Managed OTLP Configuration samples with sending queue configuration. The sending queue and batching configuration has been analysed and calculated and we provide here the final suggestions.

Why is it important?

In order to suggest the optimal configuration to our users with batching enabled

@gizas
updating motel config
8b7b695 
Signed-off-by: Andreas Gkizas <[email protected]>
@gizas gizas requested review from a team as code owners October 15, 2025 08:05
@mergify mergify bot assigned gizas Oct 15, 2025
@mergify Mergify
Copy link
Contributor

mergify bot commented Oct 15, 2025

This pull request does not have a backport label. Could you fix it @gizas? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

  • backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
  • backport-active-all is the label that automatically backports to all active branches.
  • backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
  • backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

@theletterf
Copy link
Contributor

@vigneshshanmugam @gizas This should bear backport labels to all versions where it applies (or at least all the active ones).

@gizas gizas added backport-active-9 Automated backport with mergify to all the active 9.[0-9]+ branches and removed backport-skip labels Oct 15, 2025
@gizas
Copy link
Contributor Author

gizas commented Oct 15, 2025

Thanks @theletterf . I added backport-active-9. I think those are the ones for Motel. Please correct me on this

theletterf
theletterf previously approved these changes Oct 15, 2025
View reviewed changes
Copy link
Contributor

@theletterf theletterf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. We'll also need @vigneshshanmugam approval before going ahead.

vigneshshanmugam
vigneshshanmugam requested changes Oct 15, 2025
View reviewed changes
Copy link
Member

@vigneshshanmugam vigneshshanmugam left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are just samples configuration.

This is the default configuration used by EDOT ->mOTLP -

elastic-agent/deploy/helm/edot-collector/kube-stack/managed_otlp/values.yaml

Lines 558 to 562 in a3f71c4

otlp/ingest:
endpoint: ${env:ELASTIC_OTLP_ENDPOINT}
headers:
Authorization: ApiKey ${env:ELASTIC_API_KEY}
timeout: 15s
, We need to fix that with our redommended values splitting logs vs metrics+ traces.

@gizas gizas changed the title Updating EDGE collector configuration samples for mOTLP input Updating EDOT collector configuration samples for mOTLP input Oct 16, 2025
@gizas
updating kube-stack values
808acef 
Signed-off-by: Andreas Gkizas <[email protected]>
@gizas
Copy link
Contributor Author

gizas commented Oct 16, 2025

These are just samples configuration.

This is the default configuration used by EDOT ->mOTLP -

elastic-agent/deploy/helm/edot-collector/kube-stack/managed_otlp/values.yaml

Lines 558 to 562 in a3f71c4

otlp/ingest:
endpoint: ${env:ELASTIC_OTLP_ENDPOINT}
headers:
Authorization: ApiKey ${env:ELASTIC_API_KEY}
timeout: 15s

, We need to fix that with our redommended values splitting logs vs metrics+ traces.

Thank you Vignesh. See updating kube-stack values

rogercoll
rogercoll reviewed Oct 16, 2025
View reviewed changes
Copy link
Contributor

@rogercoll rogercoll left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should apply the same changes to the "Logs essential Onboarding" configuration as well:

elastic-agent/deploy/helm/edot-collector/kube-stack/managed_otlp/logs-values.yaml

Line 243 in 808acef

otlp/ingest:

@gizas
Copy link
Contributor Author

gizas commented Oct 16, 2025

Roger thanks, done updating logs-values

@gizas
Copy link
Contributor Author

gizas commented Oct 16, 2025

@rogercoll https://buildkite.com/elastic/elastic-agent/builds/28874#0199ed58-74d0-4784-8fca-7d9b48c71892 is not happy due to make check-ci. Is there a way to update those templates?

@rogercoll
Copy link
Contributor

@rogercoll https://buildkite.com/elastic/elastic-agent/builds/28874#0199ed58-74d0-4784-8fca-7d9b48c71892 is not happy due to make check-ci. Is there a way to update those templates?

Looks like the mage Helm:UpdateAgentVersion command is the one formatting the files during the CI check. That should make the CI green 😄

vigneshshanmugam
vigneshshanmugam reviewed Oct 16, 2025
View reviewed changes
Copy link
Member

@vigneshshanmugam vigneshshanmugam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to remove batch processor settings.

@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

History

cc @gizas

vigneshshanmugam
vigneshshanmugam approved these changes Oct 17, 2025
View reviewed changes
Copy link
Member

@vigneshshanmugam vigneshshanmugam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

changelog/fragments/1760515775-batchingconfig.yaml
kind: other

# Change summary; a 80ish characters long description of the change.
summary: Updating EDOT gateway collector configuration samples for mOTLP input
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
summary: Updating EDOT gateway collector configuration samples for mOTLP input
summary: Updating EDOT gateway collector batch configuration when ingesting to mOTLP endpoint.

deploy/helm/edot-collector/kube-stack/managed_otlp/values.yaml
send_batch_max_size: 1500
batch/metrics:
# explicitly set send_batch_max_size to 0, as splitting metrics requests may cause version_conflict_engine_exception in TSDB
send_batch_max_size: 0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@carsonip Please check this once value once since we moved off from processor -> exporter batch and wanted to make sure we are not regressing with this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vigneshshanmugam vigneshshanmugam vigneshshanmugam approved these changes

@rogercoll rogercoll rogercoll approved these changes

@theletterf theletterf theletterf left review comments

Assignees

@gizas gizas

Labels

backport-active-9 Automated backport with mergify to all the active 9.[0-9]+ branches

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@gizas @theletterf @rogercoll @elasticmachine @vigneshshanmugam